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Executive Summary 



Do resources matter? This question remains controversial in the economics of education as 
many studies find no relationship between school resources and educational outcomes. Yet, 
improving educational perfonnance, especially of ‘hard to reach’ children, is a key area for 
government policy. This is particularly the case in countries like the UK and the US where 
the fact that many adults have poor basic skills is frequently attributed to people being ‘let 
down’ by the education system. 

In the UK, particular attention has been paid to schools in inner-city areas, where many 
pupils face problems of socio-economic deprivation and where there has been a concern 
about educational underachievement for many years. We study a flagship policy of the UK 
government - the Excellence in Cities (EiC) programme - which has been designed to 
address these problems. This enables us to consider not only whether this particular policy 
was successful, but also to contribute to the more general debate as to whether resources 
matter and in what circumstances. We look at the distributional impacts of resources in a way 
that is not addressed by much of the existing literature. 

Excellence in Cities was launched in 1999 in over 400 secondary schools in England and 
since then progressively increased in coverage. It is now implemented in about a third of all 
secondary schools (over 1,000 schools). The policy involves allocating resources to 
Partnerships of Local Education Authorities (LEAs) and LEA maintained schools within their 
respective regions. Since there is considerable heterogeneity in the degree of disadvantage 
and school performance within LEAs, the policy does not cover every disadvantaged or 
poorly perfonning school in England and hence it is possible to find a comparison group of 
schools outside the policy. The three core strands of EiC are as follows: Learning Mentors, to 
help students overcome educational or behaviour problems; Learning Support Units, to 
provide short-term teaching and support programmes for difficult students; and a Gifted and 
Talented programme, to provide extra support for 5-10 per cent of pupils in each school. 
Other aspects of the EiC policy are the designation of particular schools as Specialist (i.e. in 
particular subjects) or Beacon (to disseminate good practice). Schools which are successful in 
their application for such status receive significant amounts of money. Finally, EiC also 




involves the creation of City Learning Centres (to provide ICT facilities) and Education 
Action Zones (where there is an emphasis on the sharing of good practice). 

We evaluate the average impact of EiC on educational attainment and attendance at school 
over time since its introduction in 1999. Thus, we assess the extent to which the whole range 
of activities carried out as a result of EiC funding led to an improvement in important 
educational outcomes. We focus on pupil-level attainment at age 14 (the end of Key Stage 3) 
and a measure of school attendance (the percentage of half days missed). We consider 
variation in the effect of the policy according to when it was introduced to different areas; 
time since its introduction; the level of disadvantage of the school (and of pupils of different 
abilities within these schools). Our methodology is based on a comparison between outcomes 
in schools where the EiC has been in place and schools in an appropriate comparison group 
before and after the policy was introduced. Specifically, we use a difference-in-differences 
approach that is combined with statistical matching. 

Our results show a positive effect of EiC on pupil attainment in Mathematics (though not in 
English) and on school attendance. A simple assessment suggests that the policy is cost- 
effective - at least for more recent years. However, the effect of the policy is heterogeneous 
along a number of dimensions: it is stronger the longer the policy has been in place; for 
disadvantaged schools; for medium-high ability pupils within disadvantaged schools. Our 
results show that ‘resources matter’ but that it is difficult to help the ‘hard-to-reach’ using this 
level of resources (i.e. low ability children within disadvantaged schools). For such children, 
different and probably more intensive policy treatments may be required - ideally earlier in 
their schooling career. 
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Introduction 



Whether increasing resources matters for schooling outcomes, at all stages of the education 
sequence, remains controversial and at the heart of the education policy agenda of many 
countries. Part of the reason is the considerable interest in the difficult question of how to 
turn around schools that are not performing well. Part is the concern as to whether additional 
resources can be used to improve the outcomes of ‘hard to reach’ pupils who currently leave 
the education system with few or no educational qualifications. This is a particular problem 
in countries, like the UK and US, where there are long and sizable tails in the bottom end of 
the adult distribution of basic literacy and numeracy skills (OECD, 1995). The existence of 
large numbers of adults with poor basic skills in these countries (and the lack of them in other 
countries like Sweden or Germany) is frequently attributed to people being ‘failed’ or ‘let 
down’ by the education system. 

This is all the more relevant, and of general concern, because by now a large body of research 
fails to find evidence that giving more money to schools matters (see Hanushek, 2003). Only 
a small minority of papers - including the rather specific Maimonides rule paper on class 
sizes in Israel (Angrist and Lavy, 1999) and the ‘experimental’ Tennessee STAR class size 
reduction papers (Krueger, 1999; Krueger and Whitmore, 2001) - conclude that increasing 
resources matters for improving educational standards. 1 And, even then, it is often argued 
that school inputs can only matter to a limited extent when compared to the impact of family 
background and the home environment. 

Placed in this policy and research context, whether there is any scope for education policy to 
raise attainment in poorly perfonning schools becomes very important. This is true both for 
the individual pupils concerned and for the future of schools, especially if they become 
labelled as ‘failing’. In English state schools in the 1990s, there were severe problems and a 
genuine need to raise standards, especially in inner city areas (LeGrand, 1993; Machin and 
Vignoles, 2005) . Whilst the current UK government has a stated commitment to raise 

1 Recent DfES commissioned work of relevance includes Levacic et al. (2005). 

2 There are a number of aspects of poor performance or ‘under-performance’ that have been identified, which 
include both pupil and school-specific factors. For example, with regard to pupils, OFSTED (1993) identified 
underachievement that is evident even at an early age and substantial weaknesses in pupils’ oral and written 
communications. The report also remarked that these problems were substantially more prevalent amongst 
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standards in all schools, it has also devoted specific attention to the educational perfonnance 
of children attending inner city schools, especially in deprived areas. This has been justified 
in various ways, including the notion that education acquisition matters a lot for adult 

-j 

outcomes, with some implication that it may well matter even more in the future/ 

In this paper we ask the question of whether additional school resources matter. In doing so, 
we look in some detail at distributional impacts of resources in a way not addressed by much 
of the work on school resources, which tends to be more stark in its focus on giving a yes/no 
answer to the question. The approach we follow is to study a flagship policy of the UK 
government - the Excellence in Cities (EiC) programme - with an aim to shed some light on 
whether education policies that provide additional resources (directly targeted at the lower 
end of the income and education distribution) can improve educational standards. EiC is a 
well publicised policy aimed specifically at alleviating underachievement in inner city 
schools within England. In this programme, schools in disadvantaged, mainly urban, areas of 
England were given extra resources to try to improve standards. We use a study of the effects 
of the programme to question whether such a resource based programme has scope to be 
useful in turning around the fortunes of badly performing schools in the inner cities . 4 



The perceived need to turn around the fortunes of schools in deprived inner city areas is 
certainly not unique to the UK. In the US, high profile policies include school vouchers and 
Charter schools. However, unlike in the US and other countries where the school system is 
highly decentralised, the state school system is run on a national basis in the UK (albeit with 
differences between England, Wales, Scotland and Northern Ireland). Hence there is scope 
for educational policies targeted at disadvantaged groups to be formulated and assessed in a 
much more systematic manner. This is because the policies themselves are designed with the 
national system in mind, trying at the same time to isolate areas where there are particular 
problems associated with poor educational outcomes. Of course, the target schools are in 



pupils attending inner city state schools. Moreover, lack of ambition during the school years was pinpointed in 
that many pupils seemed unconvinced of the value of continuing their studies beyond the compulsory school 
leaving age. At school level there were also significant problems. These included an insufficient pace and 
challenge in teaching, and poor arrangements for learning support. Furthermore, and unlike schools in other 
parts of the country, it appeared that weaknesses in particular schools were likely to be exacerbated by poor 
links between them and other schools. 

3 See, for example, the Excellence in Schools statement that ‘the demands of the future will require that 
everyone succeeds in secondary education’ (DfEE, 1997). 

4 See Machin et al. (2004) for some early findings on EiC. 
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economically and socially deprived areas, and hence we need to pay close attention to 
selection issues in implementing our modelling approach. 

The success or otherwise of the Excellence in Cities policy is of interest in an international 
context for a number of reasons. Firstly, as discussed above, it is an example of a policy 
designed specifically for schools in disadvantaged areas - and this is also a concern for policy 
in other countries. Secondly, it is essentially a resource-based policy. As already noted, there 
are many examples of resource-based policies that have failed to have an impact on 
outcomes, which has led some to argue that such policies should be rejected in favour of 
incentive-based policies (Hanushek, 2003) whereas others argue that this conclusion is 
attributable to the poor quality of many evaluation studies (Krueger, 2003). Thirdly, the 
efficacy of area-based policies is of interest in other European countries. For example, a 
recent evaluation of such a policy in France could find no evidence for positive impacts on 
educational outcomes (Benabou el al. 2005). Finally, it has been argued that the focus for 
policy should be in early childhood interventions since it is more difficult to improve an 
individual’s abilities later in life (Carneiro and Heckman, 2003). As noted by Favy and 
Schlosser (2005), the lack on evidence on the effectiveness of high-school interventions is 
particularly important given the debate over the relative merit of early versus late childhood 
intervention. 5 

The rest of the paper is structured as follows. In Section 2, we outline the salient features of 
the Excellence in Cities policy; in Section 3, we describe our data sources and then in Section 
4 present a descriptive analysis of how outcome measures have evolved over time within 
schools exposed to EiC compared to other schools. In Section 5, we discuss the methodology 
- a difference-in-differences approach, which is combined with statistical matching. Then in 
Section 6, we present the results of the analysis - firstly, the overall impacts of the EiC policy 
on each outcome measure; and secondly a more detailed analysis for each EiC Phase where 
we examine how the effects of the policy vary over time; by the level of disadvantage in the 
school and for pupils of different abilities within these schools. In Section 7, we report the 
results of robustness tests where we examine whether the results reported in the differences- 



5 Their study is a notable exception. They evaluate the impact of remedial education of teenagers on 
achievement at high school and find strong evidence on the efficacy of augmenting instruction time for targeted 
students. 
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in-differences analysis could be attributed to the existence of pre-policy trends. In Section 8 
we outline a simple Cost-Benefit Analysis, before concluding in Section 9. 



2 Resources in English Schools and the Excellence in Cities Programme 



Excellence in Cities was launched in 1999 in over 400 secondary schools in England and 
since then progressively increased in coverage. It is now implemented in about a third of all 
secondary schools (over 1,000 schools). A complementary policy aimed at encouraging 
participation in higher education (College) was introduced to a subset of these areas in 2001. 
Furthermore, since 2001, a similar programme has been introduced for primary schools in 
another subset of these areas (on a pilot basis). 

In this paper, we evaluate the impact of the Excellence in Cities policy in secondary schools 
over the time since its introduction. It has been introduced in three main phases: Phase 1 
areas, as from September 1999; Phase 2 areas, as from September 2000 and Phase 3 areas, as 
from September 2001. It was also introduced to a small number of schools in regions outside 
of these areas in September 2001 (Cluster 1) and 2002 (Cluster 2). 6 

The Excellence in Cities (EiC) policy involves allocating resources to Partnerships of Local 
Education Authorities (LEAs) and LEA maintained schools within their respective regions. 
There are 150 LEAs in England and within the areas chosen for the three main Phases of EiC 
policy, all LEA maintained schools were affected. 7 Since there is considerable heterogeneity 
in the degree of disadvantage and school perfonnance within LEAs, the policy does not cover 
every disadvantaged or poorly performing school in England and hence it is possible to find a 
comparison group of schools outside the policy. 

Resource allocations to ‘Partnerships’ within EiC (i.e. LEAs and their schools) were largely 
based on pupil numbers and the level of disadvantage in the LEA (as measured by the 

6 These Clusters are different from the main EiC Phases because the policy has not been targeted to all schools 
within a particular Local Education Authority but only to small clusters of primary and secondary schools 
within an area. There are only a small number of secondary schools in EiC Clusters. 

7 Local Education Authorities are responsible for the strategic management of local authority education services 
including planning the supply of school places, ensuring every child has access to a suitable school place, 
intervening where a school is failing its pupils and for allocating funding to schools. 
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percentage of pupils known to be eligible for free school meals). While there was some scope 
for discretion within Partnerships about how the funding was allocated to schools, Noden et 
al. (2001) show that criteria such as school size and the level of disadvantage of the school 
were important components of the decision. Hence there is heterogeneity in how much 
funding schools receive. This varies from £50 per pupil in the more advantaged schools (in 
terms of pupils’ eligible for free school meals) to about £140 per pupil in the least advantaged 
schools. The average for all EiC schools is about £120 per pupil per year (about 4.4 per cent 
of overall per pupil expenditure). At the outset, this funding was to be allocated to specific 
strands, but over time schools have been allowed greater flexibility over how the funding is 
used. 

The three core strands of EiC are as follows: Learning Mentors, to help students overcome 
educational or behaviour problems; Learning Support Units, to provide short-term teaching 
and support programmes for difficult students; and a Gifted and Talented programme, to 
provide extra support for 5-10 per cent of pupils in each school. Other aspects of the EiC 
policy are the designation of particular schools as Specialist (i.e. in particular subjects) or 
Beacon (to disseminate good practice). Schools which are successful in their application for 

o 

such status receive significant amounts of money. Finally, EiC also involves the creation of 
City Learning Centres (to provide ICT facilities) and Education Action Zones (where there is 
an emphasis on the sharing of good practice). 

A complementary policy (Pupil Learning Credits) was introduced to a subset of the most 
disadvantaged schools (30 per cent of schools within EiC Phase 1 areas, about 230 schools) 
between September 2001 and September 2003. This involved further expenditure on pupils 
between the age of 11 and 14 years, where the aim was to provide additional learning 
opportunities for those whose social circumstances are particularly difficult. The funding per 
pupil was either £260 or £360 and schools had great flexibility over how it was used. 9 

In this paper, we evaluate the average impact of EiC on educational attainment and 
attendance at school over the time since its introduction in 1999. Thus, we assess the extent to 
which the whole range of activities carried out as a result of EiC funding led to an 

s For example, if a school is approved for Specialist status, it receives £100,000 in capital funding and £123 per 
pupil for 4 years - or longer if it is subsequently re-designated as Specialist. 

9 We consider the combined effect of EiC with this policy where relevant. 
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improvement in important educational outcomes. We focus on pupil-level attainment at age 
14 (the end of key stage 3) 10 and a measure of school-level attendance (the percentage of half 
days missed). We consider variation in the effect of the policy according to when it was 
introduced to different areas; time since its introduction; the level of disadvantage in the 
school (and on pupils of different abilities within these schools). 

Our methodology is based on a comparison between outcomes in schools where the EiC 
policy has been in place and schools in an appropriate comparison group before and after the 
policy was introduced. 



3 Data Description 



In England, compulsory education is organised around four ‘Key Stages’ from the age of 5 to 
16. 11 At the end of each ‘Key Stage’, pupils must sit national tests (which are externally set 

and marked). This analysis is based on national-level administrative records of pupil 

12 

attainment and attendance pre-policy (1999) and post-policy from 2000 to 2003. 
Attendance, or more specifically the percentage of half days missed, is only measured at 
school-level. Attainment in English and Mathematics is measured at age 14 and is available 
for every pupil in England in the National Pupil Database. The tests at age 14 mark the end of 
‘Key Stage 3’ in the National Curriculum. In each subject, marks are given for various tests 
and then converted to a ‘level’ (on a scale of 2-8). Each level is associated with a particular 
set of knowledge and skills, which is set out in the National Curriculum. The expected 
standard at age 14 is ‘level 5’ and this forms the basis of government targets. For 2004, the 



10 We focus on pupil outcomes at age 14 and not at age 16 because we do not have an early measure of ability 
(i.e. attainment at age 11) for the latter group over the time period relevant for this study. 

11 These are Key Stage 1 (at age 7); Key Stage 2 (at age 11 - the end of primary school); Key Stage 3 (at age 14) 
and Key Stage 4 (at age 16), when pupils undertake examinations for the General Certificate of Secondary 
Education (GCSEs). 

12 Years refer to the end of the school year, when tests take place. For example, 1999 refers to the 1998-1999 
school year. 

13 In English, all students set the same tests. In Mathematics, pupils can be entered for one of four different set 
of tests (or tiers), which vary in their level of difficulty. The scoring system can change over time but the level 
received by the student is supposed to be time invariant. 
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target was that 75 per cent of 14 year olds reach level 5 or above in English and 
Mathematics. 14 

The first year of national pupil-level data that has been matched up with the pupil’s prior 
attainment (at age 11) is in 1999 (i.e. the end of the school year 1998-99) - the pre -policy 
period for schools which have been longest in the EiC policy. These national data sets of 
pupil level attainment include the students’ prior attainment, date of birth, gender and codes 
for the primary and secondary schools attended. 15 School-level variables were matched up 
with these school codes using the School Performance tables and information available in the 
LEA and School Information Service (LEASIS). 16 Further details of these data sets are 
contained in Table A1 of the Data Appendix. The set of explanatory variables used in this 
analysis are listed in the notes to Table 2. We include only ‘non-special’ schools that are LEA 
maintained in the analysis. This excludes schools that exist exclusively for students with 
special needs, all independent schools and City Technology Colleges. 

Our analysis is based on a comparison of pupils in EiC schools with pupils in non-EiC 
schools before and after the policy was introduced. We start out by comparing EiC schools to 
all other LEA maintained schools in England. We then undertake a more in-depth analysis for 
schools within each Phase of EiC. The comparison group is then based on the results of 
propensity score matching in the pre-policy period (described below). First we present a 
simple descriptive analysis of the data. 



4 Initial Descriptive Analysis 



We consider the impact of the additional resources offered under the EiC programme on 
pupil achievement and on school attendance. The main outcome measures of interest are: 



14 The actual percentage of children reaching level 5 in 2004 was 71 per cent and 73 per cent in Key Stage 3 
English and Mathematics respectively. 

15 Prior attainment is measured at Key Stage 2, when students were in primary school (at age 11). 

16 It was also necessary to change school codes in the various files where these had changed over the relevant 
time period. 
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i) the probability that students attain the expected standard of “level 5” or above in 

Key Stage 3 Mathematics and English respectively; 

ii) at school level, the percentage of half days missed. 

In Tables la-lc, we show the evolution over time of these outcome measures in EiC schools, 
each group of EiC schools (i.e. Phases 1-3, Clusters 1 and 2 - which entered the EiC policy at 
different times), and non-EiC schools. For each category, we show the change in the outcome 
measure between 1999 (i.e. before EiC had been introduced in Phase 1 schools) and the most 
recent year 2003. Then in the final column, we show the ‘difference-in-differences’ estimate 
(with associated standard errors in parentheses). This is based on the difference between the 
change over time in the outcome measure in the EiC schools (or the particular group of EiC 
schools) and that in the non-EiC schools. 

It is evident from Table la-lc that if one compares all EiC schools to all non-EiC schools in 
each year the outcome measures are lower within EiC schools at each point in time. This 
reflects the relative disadvantage of EiC schools, which is why the policy was introduced in 
the first place. 17 Furthermore, there is heterogeneity within the group of EiC schools: schools 
designated under Phase 1 (together with the small number of schools in Cluster 1) have lower 
outcome measures than other schools. This also reflects their greater relative disadvantage. 
However, when one examines changes in the outcome measures over time, it is clear that 
schools in EiC areas have generally made greater progress than schools in non-EiC areas. 
Specifically, with regard to the percentage of pupils attaining level 5 or above in Mathematics 
and English, this has increased by 11.35 and 6.38 percentage points respectively in EiC 
schools and by 7.86 and 5.19 percentage points in non-EiC schools. Hence, EiC schools have 
made faster progress over the time since the policy was introduced. The ‘difference-in- 
differences’ estimate shows the relative improvement in EiC schools to have been 3.49 
percentage points (i.e. 11.35-7.86) for Mathematics and 1.19 percentage points (i.e. 6.38- 
5.19) for English. Similarly, there has been a faster decline in the percentage of half days 
missed in EiC schools. The ‘difference-in-difference’ estimate shows a higher rate of 
improvement in EiC schools of about 1 percentage point. One can consider the ‘difference-in- 
difference’ estimate for each Phase of EiC in a similar way. In general terms, the rate of 

17 See Appendix Table A2 for summary statistics of schools by EiC Phase. 
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Table 1 Initial Descriptive Analysis 



Table la. Percentage of students obtaining Level 5 or above in KS3 Mathematics 





Number 


1999 


2000 


2001 


2002 


2003 


Change 


Diff-in- 




of 












(1999- 


Diff 




schools 












2003) 


(1999- 


















2003) 


All EiC 


1009 


53.17 


56.59 


58.8 


60.06 


64.52 


11.35 


3.49 (.3) 


Phase 1 


436 


50.39 


53.81 


56.16 


57.75 


62.67 


12.28 


4.42 (.5) 


Phase 2 


310 


54.98 


58.83 


61.21 


62.36 


66.72 


11.74 


3.88 (.5) 


Phase 3 


160 


57.02 


59.96 


61.03 


62.17 


66.54 


9.52 


1.66 (.6) 


Cluster 1 


59 


49.78 


52.99 


56.60 


57.33 


59.92 


10.14 


2.28 (1.0) 


Cluster 2 


44 


56.93 


59.22 


61.34 


61.31 


65.10 


8.17 


0.31 (1.4) 


All non-EiC 


2148 


67.40 


69.88 


71.13 


71.93 


75.26 


7.86 




Table lb. Percentage of students obtaining Level 5 or above 


in KS3 English 






Number 


1999 


2000 


2001 


2002 


2003 


Change 


Diff-in- 




of 












(1999- 


Diff 




schools 












2003) 


(1999- 


















2003) 


All EiC 


1009 


57.36 


57.80 


59.26 


61.28 


63.74 


6.38 


1.19 (.5) 


Phase 1 


436 


55.70 


55.45 


57.36 


60.01 


62.35 


6.65 


1.46 (.7) 


Phase 2 


310 


58.11 


58.75 


61.35 


62.56 


65.20 


7.09 


1.90 (.8) 


Phase 3 


160 


60.66 


60.47 


61.55 


62.49 


65.49 


4.83 


-.36(1.0) 


Cluster 1 


59 


54.31 


53.26 


54.27 


59.00 


61.74 


7.43 


2.24(1.8) 


Cluster 2 


44 


59.68 


57.45 


59.99 


62.69 


62.90 


3.22 


-1.97 


















(2-0) 


All non-EiC 


2148 


67.86 


68.40 


69.04 


70.82 


73.05 


5.19 




Table lc. Percentage of half days missed (schools in 


School Performance Tables) 






Number 


1999 


2000 


2001 


2002 


2003 


Change 


Diff-in- 




of 












(1999- 


Diff 




schools 












2003) 


(1999- 


















2003) 


All EiC 


1002 


10.48 


10.10 


10.28 


9.84 


9.07 


-1.41 


-.97 (.08) 


Phase 1 


433 


10.68 


10.30 


10.44 


9.96 


9.09 


-1.59 


-1.15 (.12) 


Phase 2 


307 


10.39 


9.89 


10.13 


9.66 


8.99 


-1.40 


-.96 (.16) 


Phase 3 


160 


9.91 


9.76 


9.88 


9.62 


9.00 


-.91 


-.47 (.15) 


Cluster 1 


59 


10.89 


10.39 


10.55 


9.96 


9.14 


-1.75 


-1.31 (.36) 


Cluster 2 


44 


10.54 


10.26 


10.75 


10.23 


9.51 


-1.03 


-.59 (.28) 


All non-EiC 


2138 


8.45 


8.23 


8.70 


8.34 


8.01 


-.44 




Notes: Based on pupils in secondary schools 


in England. Standard errors in parentheses. 
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improvement has been higher in each group of EiC schools than other LEA maintained 
schools (with the exception of schools in Phase 3 and Cluster 2 for English). 

Although these results are encouraging about the potential effect of EiC policy in bringing 
about these changes, it is important to remember that EiC and non-EiC schools have very 
different characteristics that may not be fully accounted for when using a linear difference-in- 
differences estimator. The methodological approach implemented below attempts to control 
for these characteristics in a way that enables us to infer the causal effect of EiC policy on 
these outcome measures. 



5 Methodology 

Basic difference-in-differences 

Our modelling approach involves comparing pupil performance in ‘treatment’ schools 
(subject to the EiC policy) with schools in a comparison group, while attempting to take 
account of factors other than EiC that may explain any perfonnance difference. The main 
modelling difficulty is in adequately dealing with such factors. A dynamic specification for 
pupil attainment, A, for pupil i in secondary school 5 in a particular time period t can be 
written as follows: 



Ai S t - a 0 + PEiC s + PqEiC s *D t = q + yXi St + 5Z st + AA lsq _i + a t + 8i S t (1) 

where EiC is a school level dummy variable indicating whether the school is assigned to the 

EiC group of schools, X denotes pupil characteristics, Z is a set of school characteristics and 

s is an error term. The a t tenn is a set of year dummies, included to capture year on year 

differences in pupil attainment. The model also contains a lagged dependent variable 

measuring pupil attainment in an earlier time period, t-1 for the individual pupil. The main 

coefficient of interest is P q on the variable EiC s *D t = q where D t=q is one when the policy is 

1 8 

effective in the school of child i. 



18 Note that D t _ q varies between schools in EiC since schools entered the policy in different time periods. 
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In our preferred specification we also generalise (1) by adding a full set of school fixed 
effects to control for unobservables at the school level. In this case all observable fixed 
school level characteristics (like the EiC s indicator) drop out as we model within-school 
effects of EiC: 



Ai S t - a s + [3 q EiC s *Dt=q + yXi St + 8Z st + kA IS;t -i + a t + Si st 



( 2 ) 



Heterogeneous impacts 

One of the key questions is whether the effects differ by the level of deprivation of the 
school, the ability of the pupil and the length of exposure to the programme. We can provide 
some answers to this by interacting the tenn EiC s *D t =q in equation (2) with the variables of 
interest. For example, we can estimate heterogeneous effects by dividing schools into 
categories depending on the phase of EiC to which they belong. To do so we amend equation 
(2) as follows: 



Ai S t = a s + PqiEiC s *D t = q *Phasel + P q 2EiC s *D t =q*Phase2 + 
P q 3EiC s *D t =q*Phase3 + p q 4 EiC s *D t q *Clusterl + p q4 EiC s *Dt=q*Cluster2 + yXi st + (3) 

5Z st T kAi s q-i a, Si s t 



The analysis for absences is the same as that outlined above except that it is implemented at 
school-level, as pupil-level information on absences is not collected nationally. 

Statistical matching 

Some of the non-EiC schools can be quite different (on observable and unobservable 
dimensions) from the EiC schools. We thus include detailed controls for various school-level 
characteristics and prior attainment of pupils. This effectively balances the sample between 
the treatment (EiC) and the control schools. However, along the lines suggested in the 
matching literature we use the propensity score to eliminate schools from the treatment 
sample that do not compare well to any of the control schools (and vice versa). We then carry 
out the analysis on the subset of schools with sufficient common support in the treatment and 
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control samples and allowing for school fixed effects to control for unobservables. 19 
Excluded schools are those which are very different in the treatment and comparison groups 
on the basis of a whole range of pre-policy characteristics (as summarised in the propensity 
score). Further details of the approach are provided in the Data Appendix (Figures A1-A3). 
We apply this approach when analysing the effect of EiC within each EiC Phase below. 



6 Estimating the Impact of EiC 



Overall impact on pupil attainment 

In Table 2a, regression results are reported, where the dependent variable is whether the 
student attains level 5 or above in Key Stage 3 Mathematics. In column 1, controls are 
included for EiC Phase and year; in column 2, additional controls are included for the pupil’s 
prior attainment (at age 11), gender, primary school characteristics and (pre-policy) 
secondary school characteristics (i.e. equation 2a); in column 3, school fixed effects are also 
included (i.e. equation 2b). Columns 4 and 5 repeat the most detailed specification (column 
3) for boys and girls respectively. 

In columns 1 and 2, coefficients are reported for each EiC group (i.e. Phase 1, Phase 2 etc.). 
This should be interpreted at the time constant effect of belonging to that group of schools on 
the outcome measure. For example, in column 1, there is a coefficient of -.166 for Phase 1. 
The interpretation is that pupils who go to school in Phase 1 areas are less likely than other 
pupils to attain level 5 or above by 16.6 percentage points. This is also shown in the 
descriptive table (Table 1) and simply reflects the fact that schools in these areas are more 
disadvantaged. When pupil and school-level characteristics are included (in column 2), these 
coefficients become much smaller as measures of disadvantage (such as the percentage of 
pupils eligible to receive free school meals) are explicitly included in the model. These 



19 Heckman, Ichimura and Todd, 1997 demonstrate the importance of controlling for common support. Blundell 
el al, 2004, and Emmerson el al, 2004, also combine matching with difference-in-differences We conduct this 
exercise separately for schools within EiC Phase 1, Phase 2 and Phase 3. In each case, school characteristics 
from the relevant pre-policy school year are used to estimate the propensity score. 
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Table 2: Overall Impact on Pupil Attainment 



Table 2a: Probability of attaining Level 5 or above in Mathematics 
Full sample: (2003 and 1999) 





(1) Basic 
EiC*policy on; 
EiC; year 
dummies 


(2) Controls for KS2, 
gender, secondary school 
characteristics (1999); 
primary school 
characteristics 


(3) All 
controls and 
school fixed 
effects 


(4) As (3) 
Boys only 


(5) As (3) 
Girls only 


Effect of EiC 


.034 


.019 


.019 


.018 


.019 


(EiC*policyon) 


(.003) 


(.002) 


(.002) 


(.003) 


(.003) 


Phase 1 


-.166 


-.028 


— 


— 


— 




(.009) 


(.007) 








Phase 2 


-.122 


-.028 


— 


— 


— 




(.009) 


(.007) 








Phase 3 


-.113 


-.024 


— 


— 


— 




(.012) 


(.007) 








Cluster 1 


-.182 


-.028 


— 


— 


— 




(.023) 


(.003) 








Cluster 2 


-.12 


-.029 


— 


— 


— 




(.025) 


(.009) 








Sample size 


1122164 


1122164 


1122164 


567991 


554044 


Number of schools 


3157 


3157 


3157 


3014 


3030 


Table 2b: by phase of EiC 




(1) Basic 


(2) Controls for KS2, 


(3) All 


(4) As (3) 


(5) As (3) 




EiC*Phase* 


gender, secondary school 


controls and 


Boys only 


Girls only 




policy on; EiC; 


characteristics (1999); 


school fixed 








year dummies 


primary school 


effects 










characteristics 








Effect of EiC, Phase 1 


.044 


.027 


.026 


.027 


.024 


(EiC*Phasel*policyon) 


(.005) 


(.004) 


(.004) 


(.004) 


(.004) 


Effect of EiC, Phase 2 


.039 


.017 


.016 


.014 


.018 


(EiC*Phase2*policyon) 


(.005) 


(.004) 


(.004) 


(.005) 


(.005) 


Effect of EiC, Phase 3 


.017 


.010 


.011 


.010 


.012 


(EiC*Phase3 *policyon) 


(.006) 


(.005) 


(.005) 


(.006) 


(.006) 


Effect of EiC, Cluster 1 


.023 


.014 


.012 


.004 


.019 


(EiC*Clusterl *policyon) 


(.010) 


(.008) 


(.008) 


(.010) 


(.009) 


Effect of EiC, Cluster 2 


.003 


.007 


.008 


.012 


.005 


(EiC*Cluster2*policyon) 


(.014) 


(.010) 


(.009) 


(.012) 


(.011) 


Sample size 


1122164 


1122164 


1122164 


567991 


554044 


Number of schools 


3157 


3157 


3157 


3014 


3030 



Notes: robust standard errors in parentheses (clustered on secondary schools). Specifications in (2)-(5) control 
for gender, prior attainment at age 11, a year dummy and a range of variables relevant to the pupil’s secondary 
school and primary school: number of pupils; pupil-teacher ratio; percentage of pupils with special educational 
needs (with/without statement); percentage of pupils eligible for free school meals; percentage of non-white 
pupils; average performance of primary school (in terms of absences; attainment) at the time when it was 
attended by the pupil; average performance of secondary school in the pre-policy period (in terms of absences; 
attainment) dummies for the following: all boys school; all girls school; religious school; in rural area; sixth 
form (secondary); non-EiC Specialist School (secondary); grammar school (secondary); primary school type 
(infant; independent; special; other) ; missing value dummies; Interaction terms between fsm quartile of school, 
EiC status and prior attainment group of pupil (according to attainment in KS2 Mathematics). 
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coefficients drop out of the regression when school fixed effects are included (i.e. columns 3- 
5). 



The coefficient of primary interest is EiC*policyon, which is the effect of being educated in 
an EiC school on attainment in Key Stage 3 Mathematics over a time period in which the EiC 
policy was in effect. In the simplest specification (column 1), the coefficient is .034, 
corresponding to the ‘difference-in-differences’ estimate in the descriptive table above (Table 
la). 

It may be interpreted as follows: the effect of EiC was to increase the probability of attaining 
level 5 by 3.4 percentage points if one considers outcomes in the most recent year of the 
policy (2003) with the pre-policy year. Alternatively, one could say that EiC raised the 
percentage of pupils attaining level 5 or above by 3.4 percentage points. 

However, this effect declines when one includes controls to reflect the fact that schools and 
pupils in EiC areas have different characteristics to those in non-EiC areas (although it does 
not make much difference whether or not one includes school fixed effects). Hence, 
controlling for characteristics the effect of the EiC policy is to increase the probability of 
attaining level 5 or above in Key Stage 3 Mathematics by 1.9 percentage points." The effect 
is about the same for boys and girls. 

In Table 2b, we allow separate effects for each Phase of EiC. In this case, larger effects are 
shown for schools that have been in the EiC Policy for longer. Thus, the most detailed 
specification (column 3), shows that EiC increased the probability of attaining level 5 or 
above by 2.6 percentage points in EiC Phase 1 schools; 1.6 percentage points in Phase 2 
schools; 1 . 1 percentage points in Phase 3 schools; 1 .2 percentage points in Cluster 1 schools; 
and not at all for the small group of schools that came into the EiC Policy in the last year 
(Cluster 2). 



20 We have examined other outcome measures to check whether EiC only helps to push pupils over the 
government target of level 5 or above. This is not the case. EiC increases the probability that the pupil attains 
level 4 or above by 1.1 percentage points in the most detailed specification. If we treat the dependent variable as 
continuous, the coefficient in the most detailed specification if .039. In terms of standard deviations, the 
interpretation is that EiC raised attainment by .022 standard deviations (.039/1.71 1). 
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Thus, these regressions suggest that the EiC policy has had a causal impact on raising 
attainment in Mathematics in schools exposed to the policy. However, when these regressions 
are run for attainment in English, no effects are found. The positive ‘difference-in- 
differences’ estimated reported in the descriptive table (Table 1) reduces to zero, once we 
account for the fact that pupils and schools in EiC and non-EiC areas have different 
characteristics. 

Overall impact on school attendance 

In Tables 3a and 3b, analogous results are presented to those described above except that in 
this case, the data is at school-level and the dependent variable is the percentage of half days 
missed. To estimate these regressions, all variables are averaged to school-level (see notes to 
Table 2 for a list of variables included in the controls). 

As in the descriptive table (Table 1), the first column suggests that EiC policy led to a 
reduction in the percentage of half days missed by about 1 percentage point. Once controls 
are added, the regressions suggest that the effect is to reduce absences by .59 of a percentage 
point. Table 3b shows the effect of EiC on absences to vary by EiC Phase. The effect is larger 
in schools that have been in the EiC policy for longer. For the three main Phases of EiC 
(Phase 1-3), the effect is -.762, -.634 and -.349 respectively. For the smaller group of schools 
in EiC clusters, the effect is about -.56. Thus, as with attainment in Mathematics, the EiC 
policy seems to have been effective in reducing absences or alternatively, increasing pupil 
attendance. 

Heterogeneity in the effect of EiC 

The EiC programme does seem to have an effect on outcomes. In this sense increasing 
resources in a country such as the UK does seem to matter. We now wish to learn a bit more 
about who benefits most and how long it takes for such benefits to be felt. 

Allocation of funding is partly based on the proportion of pupils in receipt of free school 
meals (as discussed in Section 2). First this means that the effect at the school level is likely 
to be lower for schools with less deprived pupils. However, it may well be that there are spill- 
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over effects from the funding so that more funding outweighs the increased number of pupils 
on free school meals (FSM). 



Table 3: Overall Impact on School Attendance 



Table 3a: School-level absences: % half days missed Full sample: (2003 and 1999) 





(1) Basic 

EiC; year dummies 


(2) Controls for KS2, gender, 
secondary school characteristics 
(1999); primary school 
characteristics 


(3) All controls and 
school fixed effects 


Effect of EiC 


-.954 


-.523 


-.590 




(.086) 


(.12) 


(.193) 


Phase 1 


2.132 


-.638 


— 




(.164) 


(.265) 




Phase 2 


1.933 


-.605 


— 




(.177) 


(.260) 




Phase 3 


1.697 


-.598 


— 




(.208) 


(.267) 




Cluster 1 


2.257 


-.81 


— 




(.417) 


(.202) 




Cluster 2 


2.270 


-.373 


— 




(.458) 


(.317) 




Sample size 


6043 


6043 


6043 


Number of schools 


3068 


3068 


3068 



Table 3b: The effects of the EiC by phase of implementation 





(1) Basic 
EiC*Phase* 
policy on; EiC; 
year dummies 


(2) Controls for KS2, gender, 
secondary school characteristics 
(1999); primary school 
characteristics 


(3) All controls and 
school fixed effects 


Effect of EiC, Phase 


-1.16 


-.70 


-.762 


1 


(.119) 


(.146) 


(.229) 


Effect of EiC, Phase 


-.968 


-.524 


-.634 


2 


(.155) 


(.157) 


(.236) 


Effect of EiC, Phase 


-.482 


-.216 


-.349 


3 


(.147) 


(.173) 


(.261) 


Effect of EiC, 


-1.31 


-.620 


-.561 


Cluster 1 


(.36) 


(.297) 


(.429) 


Effect of EiC, 


-.607 


-.531 


-.559 


Cluster 2 


(.282) 


(.303) 


(.448) 


Sample size 


6043 


6043 


6043 


Number of schools 


3068 


3068 


3068 



Notes: See Table 2 for list of controls. They are computed as school-level averages. Standard errors are 
clustered by school. 
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As a result, comparing across levels of deprivation or across phases, while useful, does not 
tell us about whether extra resources matter more or less conditional on the overall level of 
deprivation. On the other hand, the comparisons relating to the effect of the programme over 
time in the same schools is informative about the importance of length of exposure and/or 
learning about how best to use the extra resources. Finally, we also compare the effects of the 
programme by pupil ability, as measured by achievement prior to the programme. This 
comparison can be informative about which type of pupil is likely to benefit most by the 
increased resources. 

The category of disadvantage is defined by the ranking of the school in terms of the 
distribution of free school meals in the pre-policy year. Schools are divided into four quartiles 
of free school meals where the lowest quartile represents a school with a very low percentage 
of pupils eligible for free school meals and the highest quartile represents schools with a 
relatively high percentage of pupils eligible for free school meals. Pupil ability is defined by 
attainment at age 11 in Mathematics (i.e. the pupil’s percentile ranking in the total 
Mathematics score at Key Stage 2). Specifically, students are divided into the following 
categories of ‘ability’: Tow’ (0-33 ld percentile); ‘medium’ (33 rd - 66 th percentile); ‘high’ 
(66 th -100 th percentile). 21 For each group of EiC schools (i.e. Phase 1, 2 and 3 respectively), 
we define a suitable set of control schools from non-EiC schools. - As explained in Section 5, 
this excludes schools in the EiC group that look very different from non-EiC schools (in 
tenns of their observable characteristics) in the pre-policy year and vice versa. The 
difference-in-differences approach is then applied to the sub-sample of schools. We report 
results using the most detailed specification (i.e. equivalent to column 3 in Tables 2a and 2b). 



In Table 4a, we show the effect of EiC policy in each year over the time since its 
introduction. There is a similarity between EiC Phases in that the policy had no impact in the 
first year after its introduction, but a significant, positive effect in the most recent year. For 
EiC Phase 1, the effect was close to 1 percentage point in the second and third year of the 
policy, and is 1.9 percentage points in the most recent year (more specifically, it led to an 



21 In Table A3 of the Data Appendix, we show the number of pupils/schools in these different categories in the 
most recent year of the EiC Policy (2003) for each EiC Phase. 

22 For Phase 1, the pre-policy year is 1999; for Phase 2 it is 2000 and for Phase 3 it is 2001. 
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increase in the probability of attaining level 5 or above in Key Stage 3 Mathematics by 1.9 
percentage points). In EiC Phase 2, there was no significant effect of the policy in the second 
year of the scheme, but is about 1 percentage point in the most recent year. Similarly, the 
effect of EiC for pupils in Phase 3 schools is 1.1 percentage point in the most recent year. It is 
not surprising to see that EiC took time to have an effect on pupil outcomes. From survey 
work undertaken at the beginning of the evaluation, it was noted that ‘for many Partnerships, 
the first year and some of the second year was a tooling up period, with resources not flowing 

O'? 

to all EiC partner schools and Strands in an immediate or uniform manner’. 

In Table 4c, we report results broken down by the level of disadvantage in the school 
(proportion of pupils with FSM). There is a similar pattern across all EiC Phases: the effect of 
the EiC policy is stronger in schools that are relatively more disadvantaged according to this 
measure. Thus, significant positive effects of EiC policy (in 2003 relative to the pre-policy 
year) are found for schools in the highest quartile of free school meals in EiC Phases 1 and 2; 
and for schools in the second-highest quartile of free school meals in EiC Phase 3. There are 
no significant effects of the policy in any other category of school. According to these results, 
EiC increased the probability of attaining level 5 or above in more disadvantaged schools by 
3.4, 2.0 and 2.4 percentage points in Phase 1, Phase 2 and Phase 3 areas respectively, if one 
compares the most recent year of the scheme (2003) with the pre-policy year. We have 
already noted that the funding allocation is partly based on the proportion of pupils in the 
school with free school meals. One should also remember that 30 percent of EiC schools 
within Phase 1 areas were also exposed to the Pupil Learning Credit policy from 2001 
onwards (i.e potentially affecting outcomes in Phase 1 areas in 2002 and 2003). Almost all of 
these schools are within the group of most disadvantaged schools (i.e. the highest quartile of 
free school meals) and in fact, are the most disadvantaged schools of this group. Hence we 
should interpret the effect for EiC Phase 1 as the combined effect of EiC and PLC policy, 
which implies that the most deprived schools received increased funding more than 
proportionately to their increased numbers of pupils on FSM. 

Finally, in Table 4d, we report the effects of EiC Policy for pupils of different abilities within 
each category of schools. For each EiC Phase, larger effects of the policy are found for pupils 



23 National Evaluation Consortium, (2001). 
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of high or medium ability than for those of lower ability - and are generally only found 
within more disadvantaged schools (i.e. schools in the highest or second-highest quartile of 
the distribution of free school meals). 24 For EiC Phase 1, the effect of EiC policy on high- 
ability pupils is to raise the probability of attaining level 5 by 4.8 percentage points in the 
most disadvantaged schools, and by 1.4 percentage points for those in the next category of 
disadvantage. The only other significant effects are observable for pupils of medium and low 
ability in the most disadvantaged category of schools, where EiC is shown to raise their 
probability of attaining level 5 or above by 3.3 percentage points and 2.6 percentage points 
respectively. A similar pattern of results is observed for all phases of the programme, as can 
be seen in the Table. 



Table 4: Heterogeneity in the Effect of EiC Across Phases, Free School Meal Eligibility 
and Ability 



Table 4a. Effects of EiC in each year (relative to pre-policy baseline in each case) 





(1) Phase 1 and non- 


(2) Phase 2 and non- 


(3) Phase 3 and non- 




EiC schools 


EiC schools 


EiC schools 


Effect of EiC, 2000 


.000 


— 


— 




(.003) 






Effect of EiC, 2001 


.008 


.005 


— 




(.004) 


(.003) 




Effect of EiC, 2002 


.007 


.005 


-.001 




(.003) 


(.003) 


(.004) 


Effect of EiC, 2003 


.019 


.010 


.011 




(.004) 


(.004) 


(.005) 


Sample size 


1756923 


1469496 


931169 


Number of schools 


1868 


1920 


1563 


Table 4b. Effects by Free School Meal Eligibility quartile 




(2003 relative to pre-policy baseline) 








(1) Phase 1 and non- 


(2) Phase 2 and non- 


(3) Phase 3 and non- 




EiC schools 


EiC schools 


EiC schools 


Effect of EiC, lowest 


-.018 


-.011 


.008 


FSM 


(.013) 


(.005) 


(.008) 


Effect of EiC, 2 nd 


-.000 


.004 


.008 


lowest FSM 


(.006) 


(.008) 


(.009) 


Effect of EiC, 2 nd 


.012 


.004 


.024 


highest FSM 


(.007) 


(.006) 


(.007) 


Effect of EiC, highest 


.034 


.020 


.001 


FSM 


(.006) 


(.006) 


(.008) 


Sample size 


699245 


727861 


616354 


Number of schools 


1868 


1919 


1563 



~ 4 The exception to this is schools in the 2 nd quartile of free schools meals that are also in EiC Phase 3. In this 
case, an effect of EiC policy is found for high ability pupils. 
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Table 4c. Effects for ability group within each FSM quartile 





(1) Phase 1 and non- 
EiC schools 


(2) Phase 2 and non- 
EiC schools 


(3) Phase 3 and non- 
EiC schools 


Effect of EiC, lowest 


-.012 


.045 


.053 


FSM, low ability 


(.040) 


(.038) 


(.025) 


Effect of EiC, lowest 


-.016 


-.037 


-.015 


FSM, medium ability 


(•021) 


(.016) 


(.022) 


Effect of EiC, lowest 


-.020 


-.009 


.007 


FSM, high ability 


(.008) 


(.004) 


(.007) 


Effect of EiC, 2 nd 


.016 


.016 


.006 


lowest FSM, low 
ability 


(.016) 


(.018) 


(.020) 


Effect of EiC, 2 nd 


-.006 


-.002 


.006 


lowest FSM, medium 
ability 


(.012) 


(•012) 


(.016) 


Effect of EiC, 2 nd 


-.005 


.003 


.017 


lowest FSM, high 
ability 


(.005) 


(.005) 


(.006) 


Effect of EiC, 2 nd 


.007 


-.003 


.022 


highest FSM, low 
ability 


(.012) 


(.010) 


(.013) 


Effect of EiC, 2 nd 


.016 


.013 


.035 


highest FSM, medium 
ability 


(.011) 


(.010) 


(.010) 


Effect of EiC, 2nd 


.014 


.008 


.025 


highest FSM, high 
ability 


(.006) 


(.004) 


(.005) 


Effect of EiC, highest 


.026 


.000 


-.001 


FSM, low ability 


(.008) 


(.008) 


(.012) 


Effect of EiC, highest 


.033 


.042 


-.013 


FSM, medium ability 


(.009) 


(.011) 


(.013) 


Effect of EiC, highest 


.048 


.034 


.029 


FSM, high ability 


(.005) 


(.005) 


(.006) 


Sample size 


699245 


727861 


616354 


Number of schools 


1868 


1919 


1563 



Notes: only most detailed specification reported (i.e. as in column 3, Table 2). See notes to Table 2 for a list of 
control variables. Standard errors are clustered by school 
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These results carry two important messages: It is quite clear given these results that an 
increase in resources can lead to improvement in the performance of pupils in some of the 
most disadvantaged schools in the country. Perhaps what may seem disappointing, from a 
policy perspective is that the benefit is almost entirely concentrated among the higher ability 
pupils. Thus the recurring theme that success builds on success (Carneiro and Heckman, 
2003; Currie, 2001) is seen again here. This points to the need for early intervention to 
improve early achievement so as not to create an “underclass” of pupils in deprived areas that 
cannot be reached by policy interventions in the later years of their schooling. 



7 A Pre-Policy Robustness Test 



The main methodological concern is the possibility that outcomes were trending upwards in 
EiC areas (and differentially to that in non-EiC areas) prior to the introduction of the EiC 
policy. If this were the case, it would undermine the argument that improvements have been 
the result of EiC policy itself (i.e. since standards were already improving before the policy 
was introduced). Unfortunately, we only have information on students’ prior ability and 
primary school attended in EiC Phase 1 areas for one year prior to the EiC policy. However, 
we have information on the outcome measure back as far as 1997. This enables us to conduct 
the analysis (albeit using a much reduced set of control variables) pooling the years 1997 to 
2003 and allowing the impact of EiC to vary by whether the policy was on or off. Similarly to 
Section 6, we estimate regressions for each EiC Phase separately for each matched sample of 
schools. We control for pupil gender, year and school fixed effects in addition to the EiC 
variables. Results are reported in Tables 5a and 5b. In the fonner, ‘policy on’ and ‘policy off 
years are aggregated for each EiC phase respectively. In the latter, the ‘EiC effect’ is reported 
in each year, where the ‘policy-on’ years are highlighted. In Table 5b, we also show the ‘EiC 
effect’ for Phase 2 and 3 when controlling for a full set of covariates for years in which this 
information is available (i.e. one ‘policy-off year for Phase 2; two ‘policy-off years for 
Phase 3). 
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Table 5: A Pre-Policy Robustness Check 



Table 5a: Evaluating the ‘EiC effect’ for the ‘policy-off and ‘policy-on’ years 



Matched sample of Matched sample of Matched sample of 

EiC Phase 1 and non- EiC Phase 2 and non- EiC Phase 3 and non- 

EiC schools EiC schools EiC schools 



‘Effect’ of EiC, 


.011 


.007 




.009 


policy-off 




(.004) 


(.004) 




(.005) 


Effect of EiC, policy- 


.026 


.030 




.019 


on 




(.005) 


(.006) 




(.006) 


Sample size 




2414474 


2503875 




2100876 


Number of schools 


1868 


1920 




1563 


P- value: 




.000 


.000 




.022 


EiC*policyoff 

EiC*policyon 


versus 










Table 5b: Evaluating the ‘EiC effect’ in each year. (‘Policy- 


on’ years are highlighted in bold) 




Matched 


Matched 


Matched 


Matched 


Matched 




sample of EiC 


sample of EiC 


sample of EiC 


sample of EiC 


sample of 




Phase 1 and 


Phase 2 and 


Phase 2 and 


Phase 3 and 


EiC Phase 3 




non-EiC 


non-EiC 


non-EiC 


non-EiC 


and non-EiC 




schools 


schools 


schools 


schools 


schools 


‘Effect’ of 


.012 


.001 





.004 





EiC, 1998 


(.004) 


(.004) 




(.005) 




‘Effect’ of 


.010 


.003 


— 


.007 


— 


EiC, 1999 


(.005) 


(.005) 




(.006) 




‘Effect’ of 


.015 


.015 


.007 


.013 


.002 


EiC, 2000 


(.005) 


(.006) 


(.003) 


(.006) 


(.004) 


‘Effect’ of 


.025 


.028 


.012 


.013 


-.003 


EiC, 2001 


(.006) 


(.006) 


(.003) 


(.006) 


(.004) 


Effect of 


.026 


.027 


.012 


.015 


-.004 


EiC, 2002 


(.006) 


(.006) 


(.004) 


(.006) 


(.005) 


Effect of 


.039 


.034 


.016 


.023 


.008 


EiC, 2003 


(.006) 


(.006) 


(.004) 


(.007) 


(.005) 


All controls 


No 


No 


Yes 


No 


Yes 


Sample size 


2414474 


2503875 


1824868 


2100876 


1528535 


Number of 


1868 


1920 


1920 


1563 


1563 



schools 



Notes: year dummies; school fixed effects and a control for pupil gender are also included. 
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Table 5a shows that the EiC effect is considerably larger in the ‘policy-on’ years than in the 
‘policy-off years. One can easily reject the hypothesis that the coefficient on the ‘policy-off 
years is equal to that of the ‘policy-on’ years and hence there is clear evidence of an 
improvement in educational standards in the EiC period. Nonetheless, there is still a small 
positive coefficient for the EiC effect in the ‘policy-off years. Further infonnation on this is 
provided in Table 5b, where the EiC effect is estimated in each year. This shows that the EiC 
effect is significantly larger from the second year of the EiC policy onwards in the case of 
Phase 1 and 3 and from the first year of the policy in the case of Phase 2. Furthermore, any 
pre-policy trends become less marked (in the case of Phase 2) or non-existent (in the case of 
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Phase 3) when additional controls are included. 

Hence, this analysis suggests that the positive effects described in the previous section can 
mainly be attributed to the effect of the EiC policy. 



8 Cost-Benefit Analysis 



We have seen that the impact of EiC was to raise attainment in Mathematics (though not in 
English) and to increase pupil attendance. To conduct an accurate Cost-Benefit Analysis, one 
would need to know how such effects translate into a range of later outcomes - for example, 
further education, probability of employment, wages, crime. There is also a possibility that 
the increase in pupil attendance has had a contemporaneous impact on crime in local areas (a 
possibility we do not explore here). Ideally, one would want to follow the children affected 
by these particular policies (and comparison groups) as they progress through school and into 
the labour market. 

In the absence of this information, we need an alternative way to estimate the potential cost- 
effectiveness of EiC policy. We have good information on the costs - the average is £120 per 
pupil per year. One relevant question is how much the average benefits in terms of exam 
attainment (i.e. Table 4a) would have to translate into higher wages for the policy to break- 

25 Note that the reason that coefficients in Table 5b, columns 3 and 5, are not the same as those reported in Table 
4a, columns 2 and 3, is that the base year is different. 1999 is the base year in the case of Table 5b; the first pre- 
policy year is the base year in the case of Table 4a. 
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even. The average rate of return to a year of schooling is fairly well accepted to be about 8 
per cent for the UK. Using the Family Resources Survey for England and Wales, we can 
obtain a wage profde (an average of weekly earnings by age, for all individuals). If pupils 
were to obtain the equivalent benefit of a whole year of education at age 14 and then started 
work at age 16, the lifetime benefit of this extra year is estimated to be about £20,000. ~ 

For a pupil in EiC Phase 2, exposed to three years of EiC policy (i.e. pupils taking 
examinations in 2003), the average benefit of EiC is to increase attainment in Mathematics by 
0.01 of a level (Table 4a). According to the National Curriculum a one level improvement 
corresponds to about 2 years of schooling. If this is true, the benefit of EiC is about 0.02 of a 
year of schooling (i.e. 0.01 x 2) - which comes to about £400 over the lifetime (i.e. 0.02 x 
£20,000). This is very similar to the cost of EiC policy (£120 x 3). Clearly, the benefit 
reduces if we average across subjects (since the policy had no effect on attainment in 
English). Furthermore the existence of a complementary resource-based policy in about half 
of the schools in EiC Phase 1 increases costs (from an average of about £360 per pupil over 3 
years to about £700 per pupil) and could fully account for the extra benefit (from 0.01 of a 
level in EiC Phase 2 to about 0.02 of a level in EiC Phase 1 in 2003). Costs of the policy 
exceed benefits if one looks at the early years of the policy in each EiC phase (where there 
seems to have been no impact on exam attaimnent). However, benefits exceed costs by a 
bigger margin if one considers pupils who only had been exposed to 2 years of the policy at a 
time when there is a positive impact on exam performance (i.e. Phase 3, 2003; Phase 1, 
2001 ). 



26 This is calculated based on the weekly earnings (E) of all individuals in the Family Resources Survey 
(2002/03) between the age of 16 and 64. The Net Present Value of an extra year of schooling at age 14 is 
calculated using the below formula. The discount rate (r) is 3.5% - the recommended discount rate in the UK 
HM Treasury Green Book ( http://greenbook.treasury.gov.uk ) 



z 



E, 

(1 + r)‘ 



*0.08 = NPV 



21 The Pupil Learning Credits Pilot Scheme (PLC) was introduced to about half of the schools in EiC Phase 1 
areas in September 2001. The existence of this policy affects our interpretation of the EiC estimates in Phase 1 
for 2002 and 2003 - benefits reflect the combined impact of EiC and PLC. The policy involved an extra £260 
per pupil for most PLC schools and an extra £360 per pupil if the school had more than 50 per cent of pupils 
eligible to receive free school meals. Together with EiC, the average cost per pupil of these policies across all 
schools in EiC Phase 1 comes to about £291 per year in 2002 and 2003. 
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This very simple analysis suggests that EiC policy breaks even - at least for more recent 
cohorts - if improvement in Key Stage 3 results corresponds to years of schooling in the way 
suggested by the National Curriculum. Even if this is way off the mark, benefits of improved 
attendance at school and higher attainment at age 14 may lead to economic benefits in the 
short and long term that we do not observe - for example, increased probability of staying on 
at school beyond compulsory school-leaving, higher probability of employment, lower 
probability of turning to crime. We hope to return to these issues in later research. 



9 Conclusion 



The issue about whether giving more resources to schools can enhance pupil performance 
remains controversial. In this paper we look at the issue by means of studying a major UK 
policy initiative - the Excellence in Cities programme - which gave more money to 
disadvantaged schools with the clear aim to alleviate underachievement and raise standards in 
inner city schools in England. Since its introduction in September 1999, it has expanded to 
cover one -third of all secondary schools and related policies have been introduced to primary 
schools and to encourage participation in higher education. 

We find that, over time, the extra resources offered by the EiC policy were effective in tenns 
of improving Mathematics achievement and in increasing school attendance. However, the 
effect of the policy is highly heterogeneous for different schools and different groups of 
pupils within schools. The beneficial effect of the policy is apparent within disadvantaged 
schools, but not within advantaged schools. This may be on account of the deliberate 
distribution of resources towards more disadvantaged schools. Within these schools, pupils of 
medium to high ability are significantly more likely to benefit than ‘hard to reach’ lower 
ability children in terms of attainment in Mathematics. Moreover, a fairly crude Cost-Benefit 
Analysis suggests that, while the EiC policy had no return in the earlier years (and costs 
therefore exceeded benefits), it is likely to (at least) break-even for more recent cohorts. In 
fact, benefits from such a policy do not need to be particularly large to generate a return when 
the average cost of the policy is only £120 per pupil per year (although there is much 
variation around that number). The question for the future is whether such gains at Key Stage 
3 and increases in pupil attendance at school manifest themselves in other outcomes such as 
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further education, labour market performance and other social benefits such as crime 
reduction. 

In conclusion, our findings show that additional resources can matter, and that education 
policies can help to turn around the fortunes of poorly perfonning inner city schools. But one 
needs to take a much more nuanced view than that adopted by much of the literature, where 
the focus is very much on offering a ‘yes or no’ answer to the question as to whether 
resources matter. Our estimates show the picture is simply more complex than this. 
Resource-based policies can show positive results, even when the resources expended are 
relatively modest, but the benefits that accrue are highly heterogeneous. Indeed, in the 
context in which we study this question, the extra resources seem less effective in securing 
achievement gains for ‘hard to reach’ children, for whom different (and probably more 
intensive and earlier) policy treatments may be required. 
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Data Appendix 





Table Al: Data sources 


Data source 


Description 


Pupil-level: 

KS2-KS3 matched data sets 


National student records of primary and secondary school 
test results from the National Pupil Database. Detailed 


1998/1999 to 2002/2003 


information available on pupil results at Secondary School 
(Key Stage 3) and Primary School (Key Stage 2). Also 
contains date of birth of student, gender, code for secondary 
and primary school attended. 


School-level: 


The LEA and School Information Service (LEASIS). This 


LEASIS 


contains annual data on all schools in England in Wales. 
Most school-level data for primary and secondary schools 
comes from this data source. 


School Performance Tables 


These annual tables exist for primary and secondary 
schools. The former contains school-level information on 
perfonnance at Key Stage 2. The latter contains information 
on performance at GCSE/GNVQ. 


‘School change’ file 


This contains information on the old and new DfES 
numbers for schools where some change took place over the 
relevant time period (such as a school merger - but often 
simply administrative changes). All relevant changes were 
made to each individual data set before merging. 
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Figure Al: Propensity Scores For EiC Phase 1 and non-EiC schools 
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Graphs by treated 



Linear prediction 



Selected for ‘matching’ specification: 

Schools with a predicted linear index of the propensity score between -2.53 and 2.32 
(EiC schools: 319; Non-EiC schools: 1549) 



Note : 

The linear index of the propensity score is computed a regression of whether the school is in 
an EiC Phase 1 area (or a non-EiC area) on pre-policy school-level characteristics. The 
propensity score is illustrated for EiC Phase 1 and non-EiC schools respectively in Figure Al 
above. EiC (non-EiC) schools are retained within the matched sample if they are within the 
same range of the propensity score as the non-EiC (EiC) schools. This amounts to trimming 
tails of the distribution. 
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Density 



Figure A2: Propensity Scores For EiC Phase 2 and non-EiC schools 




Selected for ‘matching’ specification: 

Schools with a predicted linear index of the propensity score between -2.67 and 1.66 
(EiC schools: 269; Non-EiC schools: 1651) 
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Density 



Figure A3: Propensity Scores For EiC Phase 3 and non-EiC schools 
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Graphs by treated 



Linear prediction 



Selected for ‘matching’ specification: 

Schools with a predicted linear index of the propensity score between -2.45 and .96 
(EiC schools: 145; Non-EiC schools: 1418) 
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Table A2: School Characteristics by Phase of EiC in 1999 





All EiC 


Phase 1 


Phase 2 


Phase 3 


Cluster 1 


Cluster 

2 


All non- 
EiC 


Number of schools 


1009 


436 


310 


160 


59 


44 


2148 


Religious school 


21.3 


26.6 


19.4 


17.5 


8.5 


13.6 


13.8 


School has Sixth 


47.6 


52.5 


38.1 


54.4 


37.3 


54.5 


58.9 


form 
















% Special 


2.7 


2.7 


2.6 


2.6 


3.0 


3.8 


2.7 


Educational Needs, 


(2.0) 


(2.0) 


(2.0) 


(1.7) 


(2.2) 


(2.9) 


(2.2) 


with statement 
















% Special 


19.8 


20.7 


18.3 


19.6 


20.9 


20.5 


15.9 


Educational Needs, 


(11.6) 


(12.0) 


(10-3) 


(12.0) 


(13.3) 


(11.0) 


(9.4) 


no statement 
















% eligible for Free 


31.0 


36.8 


28.0 


23.4 


30.1 


24.4 


12.1 


School Meals 


(17.1) 


(19.0) 


(14.2) 


(12.5) 


(15.7) 


(12.8) 


(8.8) 


% non-white 


24.0 


34.0 


13.2 


21.3 


22.2 


12.6 


5.7 




(28.2) 


(30.1) 


(23.6) 


(24.9) 


(25.1) 


(16.7) 


(10.8) 


Pupil-teacher ratio 


16.5 


16.4 


16.6 


16.8 


16.4 


16.1 


17.0 




(1.8) 


(2.2) 


(1.7) 


(1.3) 


(1.3) 


(1.4) 


(1.4) 


Number of pupils 


947 


935 


955 


1030 


797 


916 


925 




(328) 


(341) 


(327) 


(295) 


(236) 


(350) 


(328) 


Boys school 


7.2 


10.5 


3.9 


4.4 


10.2 


4.5 


5.4 


Girls school 


9.8 


15.6 


4.5 


5.6 


13.6 


0 


6.0 


Grammar school 


2.6 


1.8 


2.9 


1.3 


8.5 


4.5 


6.4 


% 5+ GCSE, A*-C 


36.8 


35.2 


37.6 


39.5 


35.7 


39.1 


50.1 




(19.2) 


(19.8) 


(17.9) 


(17.0) 


(24.5) 


(21.6) 


(20.0) 


% 5+ GCSE, A*-G 


86.8 


85.9 


87.2 


89.1 


85.7 


87.0 


92.2 




(9.7) 


(9.9) 


(9.8) 


(7.6) 


(11.5) 


(9.2) 


(6.5) 


% no passes at 


6.0 


6.2 


5.9 


5.3 


6.3 


7.3 


3.6 


GCSE 


(5.5) 


(5.6) 


(5.4) 


(4.4) 


(6.1) 


(6.8) 


(3.4) 


Average GCSE 
score 


33.5 

(9.4) 


32.8 

(9.6) 


33.7 

(8.8) 


34.9 

(8.3) 


32.5 

(11.7) 


34.7 

(10.8) 


39.6 

(9.2) 
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Table A3: Number of pupils/schools within each category for EiC schools in 2003 



EiC 

Phase 


FSM 

quartile 


Number 

of 

schools 


Number 

of 

students 


Students: 

low 

ability 


Students: 

medium 

ability 


Students: 

high 

ability 


Students: 

missing 

ability 


1 


Low 


24 


4219 


571 


1057 


2459 


132 


1 


2 nd lowest 


41 


8605 


1975 


2688 


3809 


133 


1 


2 nd 


68 


14415 


4540 


4671 


4927 


277 




highest 














1 


highest 


168 


28870 


11371 


8870 


6954 


1675 


2 


Low 


15 


3160 


358 


812 


1903 


87 


2 


2 nd lowest 


30 


6149 


1355 


1861 


2809 


124 


2 


2 nd 


88 


18652 


5503 


6173 


6509 


467 




highest 














2 


highest 


123 


22899 


9909 


7149 


5624 


1117 


3 


Low 


10 


2012 


315 


508 


1151 


38 


3 


2 nd lowest 


22 


4544 


1023 


1583 


1881 


57 


3 


2 nd 


49 


10262 


3463 


3357 


3190 


252 




highest 














3 


highest 


60 


11432 


4810 


3500 


2578 


544 



Note : The number of schools in each FSM quartile reflects the number of EiC schools (i.e. 
Phase 1, 2 or 3) in a particular quartile of the national free school meal distribution in 2003. 
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